Skip to content

Surface engine cooldown and classify network-level blocks on /monitor#1

Merged
zydo merged 1 commit into
mainfrom
surface-engine-cooldown
Jun 18, 2026
Merged

Surface engine cooldown and classify network-level blocks on /monitor#1
zydo merged 1 commit into
mainfrom
surface-engine-cooldown

Conversation

@zydo

@zydo zydo commented Jun 18, 2026

Copy link
Copy Markdown
Owner

The /monitor page showed Yahoo as "healthy" while it was being blocked, and gave no indication of which engines the scraper had benched. Two gaps, both fixed here.

Recognize network-level blocks: classify_engine and the scraper's cooldown tracker only treated HTTP 429/403/503 as a block, but Yahoo blocks at the TCP layer (net::ERR_CONNECTION_CLOSED) with no HTTP status, so it slipped through as healthy/degraded and was never benched. A new common/block_signals.is_network_block recognizes connection-level teardowns (not plain timeouts); it feeds both the cooldown tracker (Yahoo now gets benched instead of eating a nav timeout every cycle) and classify_engine (labelled "blocked" despite a NULL status). The metrics aggregate now carries last_error_message so the API can see it.

Surface cooldown state: the EngineCooldownTracker lives in the scraper process and the API couldn't see it, so there was no "cooling down" concept at all. The scraper now snapshots tracker state to a new engine_cooldowns table each cycle (next_probe_at as absolute UTC, since the monotonic clock can't cross processes); /metrics overlays a "cooldown" health label with a probe countdown and synthesizes rows for benched engines that produced no logs, so they stay visible instead of vanishing. A stale snapshot (scraper down) is ignored.

Docs (README, OBSERVABILITY, API_REFERENCE, BLOCK_SIGNAL_*) updated.

Summary by Sourcery

Recognize network-level connection teardowns as block signals and surface the scraper’s adaptive cooldown state on the /monitor metrics API and UI.

Bug Fixes:

  • Treat Yahoo-style connection-level failures with no HTTP status as blocked in both the scraper cooldown logic and the metrics health classification.

Enhancements:

  • Add a shared block-signal helper for detecting network-level blocks and pass error messages through the scrape metrics pipeline.
  • Snapshot per-engine cooldown state from the scraper into a new engine_cooldowns table and read it from the metrics API to expose a cooldown health state and countdown.
  • Extend the /monitor UI and styles to display a cooldown label with remaining probe time and keep benched engines visible even when they produce no scrapes.

Build:

  • Update Postgres initialization and schema management to create and truncate the engine_cooldowns table.

Documentation:

  • Document network-level block handling, the new cooldown health state, and the extended metrics response fields across README, API reference, observability, and block-signal docs.

Tests:

  • Add unit and integration tests for network-level block classification, cooldown snapshotting, engine_cooldown persistence, and metrics/monitor cooldown behavior.

The /monitor page showed Yahoo as "healthy" while it was being blocked,
and gave no indication of which engines the scraper had benched. Two
gaps, both fixed here.

Recognize network-level blocks: classify_engine and the scraper's
cooldown tracker only treated HTTP 429/403/503 as a block, but Yahoo
blocks at the TCP layer (net::ERR_CONNECTION_CLOSED) with no HTTP
status, so it slipped through as healthy/degraded and was never benched.
A new common/block_signals.is_network_block recognizes connection-level
teardowns (not plain timeouts); it feeds both the cooldown tracker (Yahoo
now gets benched instead of eating a nav timeout every cycle) and
classify_engine (labelled "blocked" despite a NULL status). The metrics
aggregate now carries last_error_message so the API can see it.

Surface cooldown state: the EngineCooldownTracker lives in the scraper
process and the API couldn't see it, so there was no "cooling down"
concept at all. The scraper now snapshots tracker state to a new
engine_cooldowns table each cycle (next_probe_at as absolute UTC, since
the monotonic clock can't cross processes); /metrics overlays a
"cooldown" health label with a probe countdown and synthesizes rows for
benched engines that produced no logs, so they stay visible instead of
vanishing. A stale snapshot (scraper down) is ignored.

Docs (README, OBSERVABILITY, API_REFERENCE, BLOCK_SIGNAL_*) updated.
@sourcery-ai

sourcery-ai Bot commented Jun 18, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds shared detection of network-level block signals and wires it into both the scraper and /metrics so Yahoo-style ERR_CONNECTION_CLOSED blocks are benched and shown as blocked, and surfaces the scraper’s live cooldown state via a new engine_cooldowns table and UI elements on /monitor.

Sequence diagram for metrics request with network blocks and cooldown overlay

sequenceDiagram
    actor User
    participant Monitor as MonitorApp
    participant API as get_metrics
    participant DB as database

    User ->> Monitor: Load /monitor
    Monitor ->> API: GET /api/v1/metrics
    API ->> DB: get_scrape_metrics(window)
    DB -->> API: overall, engines (with last_error_message)
    API ->> DB: get_engine_cooldowns()
    DB -->> API: engine_cooldowns

    loop per engine row
        API ->> API: health = classify_engine(row)
        API ->> API: is_network_block(row.last_error_message)
        API ->> API: remaining = _live_cooldown_seconds(cooldown, now)
        API ->> API: engine_metrics = _build_engine(row, cooldown)
    end

    API -->> Monitor: MetricsResponse.engines[*]<br/>health (incl. blocked/cooldown)<br/>cooldown_seconds_remaining<br/>cooldown_failures
    Monitor ->> Monitor: render health label and countdown
Loading

File-Level Changes

Change Details Files
Classify connection-level teardowns as engine blocks in both the scraper cooldown logic and /metrics health labelling.
  • Introduce common.block_signals.is_network_block with curated Chromium network error patterns indicating connection-level refusals/teardowns.
  • Use is_network_block in api/v1/metrics.classify_engine to mark engines as blocked when the last scrape failed with a network-level error despite a NULL HTTP status.
  • Extend scraper cooldown._is_block_message to treat network-level block messages as block signals in addition to existing text-based indicators.
common/block_signals.py
api/v1/metrics.py
scraper/cooldown.py
tests/test_metrics.py
tests/test_engine_cooldown.py
tests/integration/test_api_integration.py
docs/BLOCK_SIGNAL_FINDINGS.md
docs/BLOCK_SIGNAL_CHARACTERIZATION.md
README.md
Persist and expose scraper engine cooldown state so /monitor can display benched engines, including those with no recent scrapes.
  • Create engine_cooldowns table in both runtime schema setup and postgres init SQL, storing per-engine failures, next_probe_at, and updated_at.
  • Add database helpers to upsert and read engine cooldown snapshots, converting remaining_seconds from the scraper into absolute DB-based next_probe_at timestamps.
  • Extend EngineCooldownTracker with a snapshot() method returning serializable per-engine cooldown state objects.
  • Have the scraper main loop publish cooldown snapshots each successful cycle, logging but ignoring persistence failures.
common/database.py
postgres/init.sql
scraper/cooldown.py
scraper/main.py
tests/integration/conftest.py
Overlay live cooldown state onto metrics and /monitor UI, including synthesized rows and countdown display.
  • Add cooldown-related fields and logic in api/v1/metrics to compute live remaining cooldown, ignore stale snapshots, synthesize empty per-engine rows for benched engines with no logs, and emit a new cooldown health label with cooldown_failures and cooldown_seconds_remaining.
  • Update the monitor HTML/JS to render cooldown label details with a human-readable probe countdown, while keeping non-cooling engines unaffected.
  • Adjust CSS to style the new cooldown state and countdown indicator consistently with existing health statuses.
api/v1/metrics.py
api/static/monitor.html
api/static/monitor.js
api/static/styles.css
tests/test_metrics.py
tests/integration/test_api_integration.py
docs/OBSERVABILITY.md
docs/API_REFERENCE.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@zydo zydo merged commit 76bee56 into main Jun 18, 2026
1 check passed

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • In upsert_engine_cooldowns, the CASE WHEN %s > 0 currently uses the failures value ((e, f, f, max(0.0, r))), so next_probe_at is controlled by failures rather than remaining_seconds; this should likely test remaining_seconds instead so an engine with nonzero failures but zero remaining does not get an unnecessary future probe time.
  • The cooldown freshness/remaining logic mixes DB timestamps with _naive_utc_now(); since engine_cooldowns.next_probe_at/updated_at are TIMESTAMP without time zone and NOW() uses the DB server clock, it would be safer either to store these as timestamptz or to be explicit about assuming the DB runs in UTC to avoid subtle drift between API and DB clocks.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `upsert_engine_cooldowns`, the `CASE WHEN %s > 0` currently uses the `failures` value (`(e, f, f, max(0.0, r))`), so `next_probe_at` is controlled by failures rather than `remaining_seconds`; this should likely test `remaining_seconds` instead so an engine with nonzero failures but zero remaining does not get an unnecessary future probe time.
- The cooldown freshness/remaining logic mixes DB timestamps with `_naive_utc_now()`; since `engine_cooldowns.next_probe_at`/`updated_at` are `TIMESTAMP` without time zone and `NOW()` uses the DB server clock, it would be safer either to store these as `timestamptz` or to be explicit about assuming the DB runs in UTC to avoid subtle drift between API and DB clocks.

## Individual Comments

### Comment 1
<location path="common/database.py" line_range="684-693" />
<code_context>
+def upsert_engine_cooldowns(
</code_context>
<issue_to_address>
**issue (bug_risk):** Cooldown persistence uses `failures` instead of `remaining` to gate `next_probe_at`, which can keep already-expired cooldowns alive.

The `executemany` call currently passes `(e, f, f, max(0.0, r))`, so the `CASE WHEN %s > 0` is checking `failures` instead of `remaining_seconds`. For engines with `failures > 0` but `remaining_seconds <= 0`, this still sets `next_probe_at` to `NOW()`, keeping them incorrectly benched.

If the goal is for `next_probe_at` to be NULL whenever `remaining_seconds <= 0`, and only set in the future when `remaining_seconds > 0`, the CASE should use `remaining_seconds`, e.g.:

```python
a = [(e, f, max(0.0, r), max(0.0, r)) for (e, f, r) in states]
cursor.executemany(
    "INSERT INTO engine_cooldowns (engine, failures, next_probe_at, updated_at) "
    "VALUES (%s, %s, "
    "        CASE WHEN %s > 0 THEN NOW() + make_interval(secs => %s) END, "
    "        NOW()) "
    "ON CONFLICT (engine) DO UPDATE SET "
    "  failures = EXCLUDED.failures, "
    "  next_probe_at = EXCLUDED.next_probe_at, "
    "  updated_at = EXCLUDED.updated_at",
    a,
)
```

This ensures already-elapsed cooldowns are stored with `next_probe_at = NULL` and not treated as still benched.
</issue_to_address>

### Comment 2
<location path="api/static/monitor.js" line_range="114-122" />
<code_context>
             row.querySelector('.c-health').dataset.health = e.health;

+            // Cooldown: show the countdown to the next probe beside the label.
+            const cd = row.querySelector('.ecooldown');
+            if (e.cooldown_seconds_remaining != null) {
+                cd.textContent = `~${this.fmtCountdown(e.cooldown_seconds_remaining)}`;
+                const blocks = e.cooldown_failures
+                    ? `${e.cooldown_failures} consecutive block${e.cooldown_failures === 1 ? '' : 's'}; `
+                    : '';
+                cd.title = `${blocks}probing again in ~${this.fmtCountdown(e.cooldown_seconds_remaining)}`;
+            } else {
+                cd.textContent = '';
+            }
+
</code_context>
<issue_to_address>
**issue (bug_risk):** Cooldown tooltip (`title`) is never cleared when an engine stops cooling, leaving stale hover text.

In `updateEngineRow`, the `else` branch only clears `cd.textContent`, so when an engine stops cooling the previous `cd.title` remains and shows stale tooltip text. Clear the title as well:

```js
} else {
    cd.textContent = '';
    cd.title = '';
}
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread common/database.py
Comment on lines +684 to +693
def upsert_engine_cooldowns(
states: "list[tuple[str, int, float]]",
) -> None:
"""Persist the scraper's per-engine cooldown snapshot for the monitor.

``states`` is ``(engine, failures, remaining_seconds)`` per tracked engine.
``next_probe_at`` is stored as absolute UTC (``NOW() + remaining``) using the
*DB* clock so the API can derive a fresh countdown without trusting the
scraper's wall-clock; a non-cooling engine (failures == 0) stores NULL.
"""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Cooldown persistence uses failures instead of remaining to gate next_probe_at, which can keep already-expired cooldowns alive.

The executemany call currently passes (e, f, f, max(0.0, r)), so the CASE WHEN %s > 0 is checking failures instead of remaining_seconds. For engines with failures > 0 but remaining_seconds <= 0, this still sets next_probe_at to NOW(), keeping them incorrectly benched.

If the goal is for next_probe_at to be NULL whenever remaining_seconds <= 0, and only set in the future when remaining_seconds > 0, the CASE should use remaining_seconds, e.g.:

a = [(e, f, max(0.0, r), max(0.0, r)) for (e, f, r) in states]
cursor.executemany(
    "INSERT INTO engine_cooldowns (engine, failures, next_probe_at, updated_at) "
    "VALUES (%s, %s, "
    "        CASE WHEN %s > 0 THEN NOW() + make_interval(secs => %s) END, "
    "        NOW()) "
    "ON CONFLICT (engine) DO UPDATE SET "
    "  failures = EXCLUDED.failures, "
    "  next_probe_at = EXCLUDED.next_probe_at, "
    "  updated_at = EXCLUDED.updated_at",
    a,
)

This ensures already-elapsed cooldowns are stored with next_probe_at = NULL and not treated as still benched.

Comment thread api/static/monitor.js
Comment on lines +114 to +122
const cd = row.querySelector('.ecooldown');
if (e.cooldown_seconds_remaining != null) {
cd.textContent = `~${this.fmtCountdown(e.cooldown_seconds_remaining)}`;
const blocks = e.cooldown_failures
? `${e.cooldown_failures} consecutive block${e.cooldown_failures === 1 ? '' : 's'}; `
: '';
cd.title = `${blocks}probing again in ~${this.fmtCountdown(e.cooldown_seconds_remaining)}`;
} else {
cd.textContent = '';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (bug_risk): Cooldown tooltip (title) is never cleared when an engine stops cooling, leaving stale hover text.

In updateEngineRow, the else branch only clears cd.textContent, so when an engine stops cooling the previous cd.title remains and shows stale tooltip text. Clear the title as well:

} else {
    cd.textContent = '';
    cd.title = '';
}

@zydo zydo deleted the surface-engine-cooldown branch June 18, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant